Linear Regression

Probablistic Model: We start with the assumption that prior to starting a sequence of experiments we have a family of random variables with means that vary linearly with respect to some deterministic independent variable. That is, there exist an intercept β0 and a slope β1 such that for each value of the independent variable x, we have a random variable Y with mean β0 + β1x. We are then given particular values of the independent variable, say x1, x2, . . . , xn, and we let Y1, Y2, . . . , Yn be the corresponding random variables. To put this in the context of our experiments, think about each xj as a level, e.g. temperature, at which one of the experiments is conducted, and let Yj be the random variable which represents the outcome of the experiment conducted at that level. More precisely, we assume that for j = 1, 2, . . . , n, Yj = β0 + β1xj + Ej , where the “errors” Ej are independent and identically distributed (i.i.d) with a common N(0, σ) distribution. Here, the common variance σ as well as the intercept β0 and the slope β1 are all unkown. Note that Yj ∼ N(β0+β1xj , σ) and Y n ∼ N(β0 + β1xn, σ/n). Consequently, after the series of experiments are performed, our data consists of pairs (x1, y1), (x2, y2), . . . , (xn, yn), where yj is the value of Yj , the experiment performed at level xj . In this case, the values of the errors Ej , namely ε1, ε2, . . . , εn, where εj = yj − (β0 + β1xj), are referred to as the “residuals” of the sequence of experiments. So, in this context, the residuals represent the values of a sample of size n from i.i.d. N(0, σ) random variables.